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6 . A translation of the amendments to the claims under PCT Article 1 9 (3 8 U. S .C . Section 3 7 1 (c)(3 )) 
has not been transmitted for reasons indicated in section 5. 

7. A copy of the international examination report (PCT/IPEA/409) is not required as the application 
was filed with the United States Receiving Office. 

8 . Annex(es) to the international preliminary examination report is/are not required as the application 
was filed with the United Stated Receiving Office. 

9. A translation of the annexes to the international preliminary examination report is not required as 
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10. An oath or declaration of the inventor (35 U.S.C. Section 371(c)(4)) complying with 35 U.S.C. 
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amendments under PCT Article 19 that were transmitted as stated in Section 3 and/or 5; and states that 
they were reviewed by the inventor as required by 37 C.F.R. Section 1 .70. 
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required, as the application was searched by the United States International Searching Authority. 
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. DECONVOLVING FAR-FIELD IMAGES USING SCANNED PROBE DATA 



L FIELD OF THE INVENTION 

The field of the invention is the combination of scanned probe microscopic 
data with far field optical and other images in order to deconvolve these images 
beyond the diffraction limit. 

II. BACKGROUND OF THE INVENTION 

Lens based far-field imaging is limited in the resolution that it can achieve by 
the characteristics of the lens. In general, there are problems of diffraction of the 
lens, problems with aberration of the lens and problems of out-of-focus radiation. 
The latter, out-of-focus radiation problem is generally partially improved by the use 
of confocal imaging methodologies; in optics, non-linear imaging techniques are also 
useful The solution of the former diffraction and aberration problems is partially 
addressed by measuring the point spread function of the lens and then using 
computer deconvolution to remove these effects from the image. Even the latter out- 
of-focus problem can be addressed without confocal or non-linear imaging by 
considering both the in-focus and the out-of-focus point spread function and using 
deconvolution routines to try and eliminate these effects, Numerous algorithms have 
been devised to address these problems of computer deconvolution of far-field 
imaging data, but none are completely successful and none of them have the ability to 
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carry the far-field image to the realm beyond the diffraction limit as defined, for 
example, by the Rayleigh criterion, which is approximately V% of the wavelength of 
the radiation that is being used. For visible 500 nm light this is 250 nm. 



In terms of deconvolution algorithms, a powerful mathematical approach is 
based on the use of constraints. For example, in deconvolving a far-field image a 
good constraint would be to define with high precision the cell membrane of a cell 
that is stained with a dye and is being imaged by a lens. By precisely defining the 
position of a cell membrane or a portion of the cell membrane, it is possible precisely 
to define where the staining in the image is confined and beyond which point or 
points there is no staining and its associated optical phenomenon. Such a constraint 
would give many deconvolution algorithms a powerful advantage. Nonetheless, even 
though the idea is mathematically a powerful concept [Carrington et al, Science 268, 
1483 (1995)], it is seriously limited in far-field optics by the inability to obtain a 
constraint that is better than the optical resolution. 



III. STATE OF PRIOR ART 

No one has previously attempted to incorporate near-field optical data and 
other scanned probe microscope data such as that which is obtained from atomic 
force microscopy to the problem of providing constraints in the deconvolution of far- 
field optical and other far-field imaging techniques. 



IV. SUMMARY OF THE INVENTION 
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In accordance with the present invention, a method for deconvolving far-field 
optical images beyond the diffraction limit includes the use of near-field optical and 
other scanned probe imaging data to provide powerful and new constraints for the 
deconvolution of far-field data sets. Near-field data, such as that which can be 
obtained from atomic force microscopy on a region of the far-field data set in an 
integrated and inter-digitate way, is used to produce resolutions beyond the 
diffraction limit of the lens that is being used. In the case of non-linear optical 
imaging or other microscopies, resolutions beyond that which is achievable with these 
microscopies can be obtained. 



V. BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing, and additional objects, features and advantages of the present 
invention will become apparent to those of skill in the art from the following detailed 
description of a preferred embodiment thereof, taken with the accompanying 
drawings, in which: 

Fig. 1A illustrates a first image of a model object, a second image of the object 
imaged by a lens having a known point spread function, a third, or sampled image 
recorded by a CCD device, and a fourth image of a deconvolution of the sampled 
image without constraints; 

Fig. IB is a graphical illustration of the deconvolution of Fig. 1A; 

Fig. 2A illustrates the same images as Fig. 1 A, wherein the deconvolution is 
performed with constraints obtained from near-field Imaging data; and 

Fig. 2B is a graphical illustration of the deconvolution of Fig. 2A. 
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VI, DESCRIPTION OF THE INVENTION 

The present invention incorporates data that has never been incorporated 

previously to resolve issues and problems in far-field imaging. This data comes from 

near-field optical microscopy and its scanned probe cousins, such as atomic force 

imaging (AFM), and presents the far-field microscopist with constraints that will 

dramatically improve the far-field imaging of all forms of far-field microscopy. These 

improvements are available for both linear and non-linear optical microscopy and 

even those microscopies that use particles rather than electromagnetic radiation. The 

invention also allows for using one form of scanned probe microscopy to deconvolve 

another, to thereby improve the resolution of a scanned probe microscope before this 

data is vised in the present invention, as described below. 

In accordance with the invention, it is essential to obtain near-field optical or 
other scanned probe imaging data in a way that is fully integrated with a far-field 
data set that is to be improved. In one embodiment of this invention in which far- 
field optical microscopic data Is to be deconvolved, one useful approach to achieving 
the required full integration of the data sets is to use a charge coupled device (CCD) 
to record the far-field data that corresponds to an associated scanned probe pixel. 
Furthermore, it is additionally useful if the scanned probe data sets that are to be 
used in the deconvolution are obtained in simultaneous channels. This can be done, 
for example, with a tip that is multifunctional such as a tip that is both a 
subwavelength light source and an AFM sensor [K. Lieberman, et aL, Rev. ScL Instr. 
67, 3567 (1996)] that can be used in contact or near-contact with a surface of a 
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specimen that is being imaged. 
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However, even before the far and near-field imaging process is begun, lens 
images of a subwavelength light source of known dimension are recorded on a CCD. 
These images, both in-focus and out-of-focus, are used to obtain a measure of the lens 
point spread function (PSF), even with perturbation of the object, or sample, being 
imaged- This PSF is the lens function that convolutes with the functional 
representation of a sample to give the blurred image that is the far-field image with 
its associated diffraction and other problems mentioned in the Background section, 
above. Alternately, the convolution effect of the lens can also be determined if a 
known high resolution sample is imaged and the error between the real and the ideal 
image is represented as a blurring function introduced by the lens. Obviously, if the 
imaging task is fluorescence then the high resolution test object will have to be 
similarly fluorescent. Thus, the first task in this method is to determine the PSF of 
the lens that is to be used to image the desired object, or sample. 

The next step is to record with the CCD the far-field image of the object. 
Subsequently, super-resolution optical data is recorded for specified points on the 
object surface. An example of such data can include defining an exact point at which 
the optical contrast in an object terminates; i.e., defining the edge of an object to 
much better than the optical resolution if the far-field image is an optical image or 
defining the x, y and z point, or voxel, at which there is a contrast change, and 
relating this point to another point of contrast change in the sample. The two points 



WO 00/47978 PCT/USOO/40002 

of contrast change in the sample could, for example, be at different planes as defined 
by the lens in the far-field, and the near-field scanned probe data, obtained through a 
multifunctional AFM sensor tip and simultaneously recorded, could provide not only 
the x-y separation of the two points but also the Z separation at a resolution that is 
better than any optical approach such as confocal microscopy. 

The exemplary constraints listed above, or for that matter any constraints from 
scanned probe technology, have never been used in this cross-feitilization mode with 
far-field optical microscopy or, for that matter, any far-field microscopic technique. 
In addition, such cross-fertilization has not been used between scanned probe 
techniques, i.e. to use near-field optical data as a constraint at deconvolving near-field 
atomic force microscopy data, or the reverse. With regard to this latter mode of 
deconvolution there is quite a bit of synergism between, for example, the near-field 
optical and the near-field AFM since the functional dependence of the decay of the 
effect, as a function of probe sample separation, is near-exponential for the near-field 
optical and occurs over a much shorter distance for the simultaneously recorded AFM 
technique. In essence, then, the "scanned probe microscopy synergism can first be 
used to improve the near-field scanned probe microscopy data and then that data can 
be applied as a constraint to the far-field microscopy deconvolution in question. At 
this point, it is also important to note that the order of the procedures listed in this 
section is not a critical part of the invention; any combination of the order of the 
steps or partial combination of steps constitutes this invention. For example, if the 
near-field optical data is not used to deconvolve to higher resolution the atomic force 
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microscopy data, it, and the atomic force microscopy data could still be used to 
deconvolve the far-field data. 



The CCD mentioned above is a most useful method to obtain the digitized 
image of the far-field, but this can also be accomplished with confocal microscopy. In 
the case of far-field optical microscopy it should be noted that there could be 
innovative ways to record the confocal data without any confocal aperture. For 
example, a film of material that produces a non-linear optical signal known as second 
harmonic generation (SHG) can be used as part of this invention to record a confocal 
image. In this case the light from the plane of focus is focused by the lens onto the 
film, such as a plastic film of purple membrane that.produces SHG [Z. Chen et aL, 
Applied Optics 30, 5188 (1991)], with an intensity that is higher than from any other 
plane in the sample that is being focused by the lens. A film that would produce a 
second harmonic signal only where there is a point of light from the plane of focus in 
the sample could be used to replace the confocal pinhole. Such a film would act as a 
parallel filter for light from the plane of focus in the sample. This could be used 
together with ah appropriate filter after the film to remove the fundamental 
wavelength that was illuminating the sample and to pass only the SHG to the 
detector which could be a CCD rather than the single channel detector that is 
normally part of a confocal set-up. This could be done with SHG or other non-linear 
optically active films. 



VII. ADVANTAGES OVER PRIOR ART 
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Scanned probe microscopy data has not been used as a constraint in 
mathematical constraint algorithms to deconvolve far-field optical images. In 
addition, the use from multi functional scanned probe microscopy of one parameter, 
such as near-field optical data, to deconvolve another parameter such as atomic force 
microscopy data has also not been applied. The advantage over prior art arises from 
the increase in spatial resolution that this approach achieves. 



VIIL APPLICATIONS 



Methodologies for increased spatial resolution always open new doors in 
science and technology as evidenced by the revolution that was caused by the 
introduction of the electron microscope. 

To test the essence of this invention a calculation has been performed on a 
model far-field optical data set, as illustrated in Figs. 1A, IB, 2A and 2B, to which 
reference is now made. In Figure 1 A there are four images, one In each of four 
horizontal rows.- At the right-hand end of each row is an intensity legend for 
comparison. Starting from the top of Fig. 1 A, the first row of the figure represents a 
model object which has a dimension that cannot be resolved optically; for example, 
an object having a dimension of about 0.2 micron. As may be seen in the first row, 
the object has a first sharp light-to-dark transition point at its left-hand edge, a 
second, dark-to light transition near the center, a third, light-to-dark transition to the 
right of the second transition, and a fourth, dark-to-light transition at the left-hand 
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edge of the model object. When the object is imaged by a lens with a known point 
spread function, each of the transition points on the object is blurred because the 
dimensions are too small to be resolved, and the blurred image, which is the second 
row from the top in Figure 1, results. When this image is recorded by a CCD there is 
further blurring due to the pixel character of the CCD, as illustrated in the third row. 
The last image, row 4 in this figure, is produced by processing the image of row 3 
with a standard deconvolution algorithm without the imposition of the type of 
constraints that are central to this invention, in which a new approach to provide 
constraints is described. Figure IB is a chart illustrating the intensity variations 
produced by the model object and by the restored (deconvolved) object. 

In Figure 2 A the same object, lens, and CCD are used, and rows 1-4 illustrate 
the same object and images as described above with respect to Fig, 1A, except that in 
the deconvolution algorithm four points are given the resolution of near-field optics. 
These points are, in going from left to right in the model object illustrated in the top 
row of Fig. 2A, the first, second, third and fourth alterations in contrast, as described 
above with respect to Fig. 1A. The results of using such constraints is seen in the 
vastly improved quality of the deconvolved image in the bottom row of that Figure. 

Although the invention has been described in terms of preferred embodiments, 
it will be understood that numerous variations and modifications may be made 
without departing from the true spirit and scope thereof as set forth in the following 
claims. 
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What is claimed is: 

1. A method for deconvolving far-field optical microscopic images to a level of 
resolution that has never been achieved previously by introducing scanned probe 
microscopy data in an interdigitated, integrated fashion with confocal or charge 
couple device imaging fashion such that every pixel or voxel from the super-resolution 
scanned probe microscopy data set is direcdy related to an image pixel or voxel in the 
far-field data set and then using the scanned probe data as a mathematical constraint 
to deconvolve the far-field image. 

2. A method that uses multifunctional scanned probe microscopy data so that one 
scanned probe microscopy parameter can be deconvolved by integrating data from 
another such as, but not excluding other combinations, deconvolving an atomic force 
microscopy image with a near-field optical microscopy image or vice versa. 

3. The method as in claim 1 in which the procedure in claim 2 is used in combination 
with the method in claim 1. 

4. A method to obtain the point spread function, by using a known high resolution 
sample either fluorescent or non-fluorescent, which is imaged and the error between 
the real and the ideal image is represented as a blurring function introduced by the 
lens and is representative of an accurate point spread function. 

5. A method to obtain a digitized image of the optical far-field in addition to simple 
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confocal or charge coupled device imaging that can be used for integrated near-field 
and far-field imaging that uses a film of material that can produce a non-linear optical 
signal known as second harmonic generation in which light from the plane of focus of 
the lens would be focused by the far-field lens onto a film, such as plastic purple 
membrane films, with an intensity that is higher than from any other plane in the 
sample that is being focused by the lens and thus such a film could be used to replace 
the confocal pinhole in confocal imaging by using a film that would light up only 
where there is a point of light from the plane of focus in the sample and thus such a 
film would act as a parallel filter for light from the plane of focus in the sample with 
this light being imaged through a filter for the fundamental illuminating frequency 
onto a charge coupled device. 

6. A device that collects in an integrated, correctable fashion that as a result allows 
for deconvolving far-field optical microscopic images to a level of resolution that has 
never been achieved previously by introducing scanned probe microscopy data in an 
interdigitated, integrated fashion with confocal or charge couple device imaging such 
that every pixel or voxel from the super-resolution scanned probe microscopy data set 
is directly related to an image pixel or voxel in the far-field data set and then using 
the scanned probe data as a mathematical constraint to deconvolve the far-field 
image. 

7* A device that collects in an appropriate integrated fashion multifunctional scanned 
probe microscopy data so that one scanned probe microscopy parameter can be 



11 



WO 00/47978 PCT/US00/40002 

deconvolved by integrating data from another such as, but not excluding other 
combinations, deconvolving an atomic force microscopy image with a near-field 
optical microscopy image or vice versa. 

8. A device as in claim 5 that also has the capabilities of claim 6. 

9. A device to obtain the point spread function, by using a known high resolution 
sample either fluorescent or non-fluorescent, which is imaged and the error between 
the real and the ideal image is represented as a blurring function introduced by the 
lens and is representative of an accurate point spread function. 

10. A device to obtain a digitized image of the optical far-field in addition to simple 
confocal or charge coupled device imaging that can be used for integrated near-field 
and far-field imaging that uses a film of material that can produce a non-linear optical 
signal known as second harmonic generation in which light from the plane of focus of 
the lens would be focused by the far-field lens onto a film such as plastic purple 
membrane films, with an intensity that is higher than from any other plane in the 
sample that is being focused by the lens and thus such a film could be used to replace 
the confocal pinhole in confocal imaging by using a film that would light up only 
where there is a point of light from the plane of focus in the sample and thus such a 
film would act as a parallel filter for light from the plane of focus in the sample with 
this light being imaged through a filter for the fundamental illuminating frequency 
onto a charge coupled device. 
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